The present application is concerned with upsampling and signal enhancement such as in the field of image/video coding.
Image upsampling refers to generating a High Resolution (HR) image from an input Low Resolution (LR) image. This task has regained attention because images/videos are being viewed on displays of varying sizes, like mobile phones, tablets, laptops, PCs, etc. For example, the content for a 1920×1080 display may be available only in a 1280×720 format and needs to be upsampled. More recently, 4K displays are becoming popular and content with a lower resolution (e.g., 1920×1080) may have to be displayed.
Image upsampling is also referred to as image interpolation, upscaling, resizing, resampling, super-resolution, etc. Many established methods are available for achieving upsampling, e.g., FIR filtering using bilinear filter (2-tap) or bicubic filter (4-tap) is popularly employed due to the ease of implementation. These techniques may cause several artifacts, most commonly, blurring of the resulting HR image. The main purpose of this invention is to recover sharp edges and textures, while reducing blurring, ringing, aliasing or other visual artifacts. For videos, there is an additional requirement to maintain the temporal coherence and to avoid picture-to-picture flickering during playback.
Image/video upsampling finds many applications in image processing, computer vision and graphics, such as compression, editing, surveillance, and texture mapping. It is vital for image browsing and video playback software. Details synthesis in image upsampling can also be used as a tool for scalable video coding. Details synthesis can also be used without upsampling, e.g., as a loop filter or post filter stage in a video coding context.
Signal processing theory for band-limited signals advocates sampling higher than the Nyquist rate and a sinc interpolation [Shannon1949, Unser2000]. The assumption of band-limitedness does not hold for most images, due to the existence of sharp edges. However, conventional schemes adhere to this philosophy and approximate the ideal low pass filter to produce acceptable results for many practical applications. Techniques like bilinear, bi-cubic interpolation, etc., are some popular examples that have low computational complexity. Extending the sampling theory to shift-invariant spaces without bandlimit constraints has led to a generalized interpolation framework, e.g., B-spline [Unser1999], MOMS interpolation [Blu2001] that provide improvements in image quality for a given support of basis functions. However, these linear models cannot capture the fast evolving statistics around edges. Increasing the degree of basis functions in these linear models help to capture higher order statistics but result in longer effective support in the spatial domain and hence produce artifacts like ringing around edges.
To improve linear models, directional interpolation schemes have been proposed that perform interpolation along the edge directions, e.g., NEDI [Li2001]. It is achieved through computing local covariances in the input image and using them to adapt the interpolation at high resolution, so that the support of the interpolator is along the edges. However, the resulting images still show some artifacts. The iterative back-projection [Irani1991] technique improves image interpolation when the downsampling process is known. Its basic idea is that the reconstructed HR image from the LR image should produce the same observed LR image if passing it through the same blurring and downsampling process. However, the downsampling filter may not be known in many cases, or the input image may be camera captured, where the optical anti-alias filter used within the sampling system is not known during the subsequent image processing stages. Therefore, it is desirable to design a method that does not rely directly on the downsampling process.
Upsampling an image is one form of image enhancement. Image enhancement aims at improving the quality of an image. In other words, image enhancement aims at reversing, at least partially, the quality degradation an image may have been subject to by, for example, lossy coding.